Intelligent Prediction of An Expected Value of User Conversion

ABSTRACT

Highly user-specific data is used to calculate user intent to make a purchase and the value of such a purchase. User activity and information is aggregated, per user, for a set window of time and real-time data on recent site behavior is obtained. Aggregated and/or real-time data is considered by a predictive intent model (calculating the probability that the user will make a purchase) and a predictive value model (calculating the expected revenue such a purchase may generate). Weights, specific to each model, are assigned to predictor features tracked in the aggregated and/or real-time user data. The most highly-weighted features of the intent model relate to users&#39; viewing history, and the most highly-weighted features of the value model relate to price and market. By these means, a user conversion value can be obtained, guiding the application of user acquisition strategies for different home sharing markets.

RELATED ART

Web-based search services, social media, and other user-facing contentdistributors and marketing channels may obtain revenue by displaying toa user, alongside content such as relevant search results,advertisements or other marketing media relevant to a user and/or theirsearched-for keywords. Such search services may charge an advertiser perimpression (view) or per click-through (follow to advertiser's site) ofan advertisement, a flat per-bid price, or other type of charge fordisplaying the advertiser's advertisement to users. Due to the costs ofadvertisement (whether monetary or resource cost), advertisers must makedecisions about which advertisements and/or marketing targets to pursue.

Traditionally, advertisers manage their marketing target audience,budgets, and user acquisition strategies though an intent-based systemwhere advertisers make guesses or inferences as to user intent in orderto select, among other things, the best channels and target groups towhom to send their advertisements. Such guesses are generally basedsolely on the advertiser's intuition and market knowledge, requiringhuman intervention and intelligence.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure can be better understood with reference to the followingdrawings. The elements of the drawings are not necessarily to scalerelative to each other, emphasis instead being placed upon clearlyillustrating the principles of the disclosure. Furthermore, likereference numerals designate corresponding parts throughout the severalviews.

FIG. 1 is a diagram of an environment configured to determine intent andvalue-based marketing return in accordance with some embodiments of thepresent disclosure.

FIG. 2 is a block diagram of certain components of a data modelingserver in accordance with some embodiments of the present disclosure.

FIG. 3 is a block diagram illustrating an exemplary data flow through adata modeling server in accordance with some embodiments of the presentdisclosure.

FIG. 4A-4F illustrate gradient descent of an exemplary linear predictivemodel in accordance with some embodiments of the present disclosure.

FIGS. 5A and 5B are charts illustrating error calculation of anexemplary linear regression predictive model in accordance with someembodiments of the present disclosure.

FIG. 6 is a flowchart of an exemplary process to generate a user-levelbid/cost value in accordance with some embodiments of the presentdisclosure.

In the figures, the left-most digit(s) of a reference number identifiesthe figure in which the reference number first appears. The use of thesame reference numbers in different figures indicates similar oridentical items or features. Moreover, multiple instances of the samepart are designated by a common prefix separated from the instancenumber by a dash. The drawings are not to scale.

DETAILED DESCRIPTION

Embodiments of the present disclosure generally pertain to systems andmethods for applying machine learning methods to predict in real timeeach site visitor's (user) likelihood for conversion (making a purchase)and potential value or revenue of that conversion, within a predefinedconversion window. The quantitative results can be used to calculate areturn on investment (ROI) for a marketing activity or campaign to thatuser, and can be used in optimizing various user acquisition channels. Auser acquisition channel may include any program or method to acquirenew or repeat customers, such as displaying ads (e.g., on Facebook,Youtube or other social media platforms), Google Display Network, GoogleKeywords, promotional emails/push notifications, referrals, landingpages, and the like.

A system in accordance with an exemplary embodiment of the presentdisclosure includes an intent model to predict a user's likelihood topurchase and a value model to predict the value of that purchase. Theintent model and the value model are positioned to access website serverlogs, other raw data sources, and aggregated data sources storinginformation about the user, their activity, and the like, which are usedto continually train and update the models. The trained models can befed aggregated and real-time data to reach the above-mentionedpredictions.

In one embodiment, the intent model may use logistic regression toassign weights to predictor features tracked in the aggregated and/orreal-time user data. The most important (and therefore most highlyweighted) features for the intent model are those related to thefrequency of a user's actions on a website, such as a user's accounthistory, same-day site behavior, past bookings/visits, frequencies ofviews, etc. In one embodiment, the value model may use linearregression, assigning different (or potentially different) weights tothe same set of features as the intent model. The most important (andtherefore most highly weighted) features for the value model are thoserelated to destination city, median price, or other price-basedfeatures. The value model optimizes such that an absolute differencebetween the predicted value of booking and the actual value of bookingis as small as possible, determining weights for the features tominimize this difference. The result of either model may be variouslyoptimized to meet certain metrics. For instance, marketing efforts mayseek to optimize a target number of bookings or a return on investment(ROI) for a marketing campaign. By use of the intent model and valuemodel, an objective user conversion value can be obtained, guiding theapplication of user acquisition strategies for different markets,including, e.g., home sharing markets.

One type of exemplary system is an online reservation system thatdisplays to users listings of properties, such as houses, condominiums,rooms, apartments, lots, and other real estate, that an owner/manager(referred to hereinafter as a “host”) may offer to the user (referred tohereinafter as a “guest”) for reservation (sometimes referred to as a“booking” or “rental”) for a specified time period (e.g., a day, week,month, or other period of interest). The online reservation systemdisplays these reservable properties to guests as property “listings,”which contain information such as price ranges, dates, number ofbedrooms, and/or other factors that may be of interest to guests inmaking a reservation. In one embodiment, when a guest books a property,the online reservation system may allow for intake, from the host, of abooking fee, an initial setup fee, a recurring fee, no fee, and/or anyother appropriate value. In some instances, the system may also handleand/or facilitate one or more financial transactions relating to thepurchase or booking of that property, and may receive a fee in relationthereto. In some such embodiments, any single property may be bookedonly once (or a limited number of times) for a certain time period. Insuch embodiments involving a home sharing market, the conversion windowis long, frequency of conversion is low, and seasonality effects bothfactors. Additionally, the product inventory (of reservable properties)consists of unique products. Accordingly, features such as listingavailabilities, market demand and supply are important in a marketanalysis.

Ideally, advertisers would manage marketing, budgeting, useracquisition, and the like based on determinations of whichadvertisements are the most valuable in acquiring users. In conventionalpractices, marketing efforts develop user intent-based budget allocationbased on rules developed from the advertiser's own intuition and domainknowledge. For instance, traditionally, a marketer might classify usersinto buckets, such as active/dormant users, or based on their frequencyof logic, and then perform an ad hoc analysis thereon. However, as thefactors impacting user acquisition scale in complexity (and as themarket becomes increasingly aware of intent-related correlativefactors), a rule-based system becomes increasingly more cumbersome tomanage and increasingly less optimal. More specifically, as user baseand complexity grows, an increasing number of “cuts” or classificationsmust be made to gain awareness and understanding about the user base,making analysis cumbersome and unfeasible. In addition, in circumstanceswhere the ultimate goal of a marketing effort is to maximize revenue, asolely intent-based system is deficient, as it does not factor in arelevant value analysis. There is therefore a need for a value-basedsystem that combines both travel intent and booking value in order tobetter understand the customers.

By virtue of the systems and methods described herein, an objective userconversion value can be obtained based on highly user-specificinformation including actual historic user activity, guiding theapplication of user acquisition strategies for different marketsincluding home sharing markets. Having user-specific data availableallows an advertiser to more efficiently select an advertisementcampaign to meet their strategic goals, optimizing for targets such asreturn on investment of advertising dollars or market penetration with alower chance of error or inefficacy. The systems described herein arehighly personalized to a user visiting the advertiser's site, as thesystems take in as a basis for their analysis each visitor's past siteactivities, account history and/or demographic information.Additionally, the systems can be easily extended to optimize for othermetrics simply by changing the target metrics, and using the sametrained machine learning models.

FIG. 1 illustrates an environment 100 that includes one or more of userdevices (referred to individually or collectively as user devices 150),which may take a variety of forms, each connected to a network 130.Network 130 may comprise one or more network types, such as a wide areanetwork (such as the Internet), a local area network (such as anintranet), a cellular network or another type of wireless network, suchas Wi-Fi, Bluetooth, Bluetooth Low Energy, and/or other close-rangewireless communications, a wired network, such as fiber optics andEthernet, or any other such network, or any combination thereof. In someembodiments, the network 130 may be the Internet and payment andinformation may be communicated between system components in anencrypted format such by a transport layer security (TLS) or securesocket layer (SSL) protocol. In addition, when the network 130 is theInternet, the components of the environment 100 may use the transmissioncontrol protocol/Internet protocol (TCP/IP) for communication. A userdevice 150 may include, among other things, one or more computingdevices, such as a smartphone or other hand-held device (e.g., a tabletor reader), desktop computer, laptop computer, touchscreen device, orany other appropriate device capable of receiving, processing, andoutputting digital information via network 130. Some user devices 150may allow a user to transmit and receive data over network 130 via a webbrowser 110 or one or more applications 120. While FIG. 1 illustrates aone-to-one correspondence between a user device 150 and one of a webbrowser 110 or app 120, the embodiments are not so limited and anyconfiguration may be possible.

In some embodiments, the user device 150 may initially transmit data toand receive data from one or more web servers 160 or one or morethird-party server(s) 180. Data modeling server(s) 170, using data fromweb server 160 and third-party server 180, may calculate scorescorresponding to user intent and/or the value of a property listing,marketing campaign, or the like, allowing for evaluation of marketingplans. In some embodiments, the components and/or functions of datamodeling server 170 may be integral with, or executed on, web server160. In other embodiments, the data modeling server 170 may beimplemented discretely from, or redundant with, the web server 160. Insome embodiments, the data modeling server 170 and the web server 160are owned and/or operated by the same entity. In an exemplaryembodiment, the data modeling server 170 is connected to the sameinternal network as the web server 160. Further, while FIG. 1illustrates only one web server 160, one data modeling server 170, andone third party server 180, it may be understood that any number of suchservers may be variously configured in different embodiments, suchservers being managed by any number of different entities.

FIG. 2 depicts an example schematic diagram of certain components ofdata modeling server 170 in accordance with some embodiments. The datamodeling server 170 includes processing elements 260, memory 210, andnetwork interface 280, which components connect through a localinterface 270 that can include at least one system bus. The networkinterface 280 can interface with the network 130. In some embodiments,network interface 280 may use a communication logic (not shown) thatincludes one or more application programming interfaces (APIs) stored inmemory 210 or another storage location, to transmit data over network130. The processing element 260 may include one or more microprocessors,central processing units (CPU), digital signal processors, graphicsprocessing units, and/or any other specialized processor, combination ofprocessors, or other circuitry that communicates to and drives the otherelements via interface 270. In some embodiments, the processing of datamodeling server 170 is not limited to being performed by a processingelement 260, but instead, any portion of processing in support of thefunction of the data modeling server 170 may be distributed over one ormore computer systems that may be remotely located. In some embodiments,the processor 260 may comprise an artificial neural network or othertype of configuration for performing machine learning functions based oninstructions stored in memory 210.

Memory 210 may refer to any suitable storage medium, either volatile andnon-volatile (e.g., RAM, ROM, EPROM, EEPROM, SRAM, flash memory, disksor optical storage, magnetic storage, or any other tangible ornon-transitory medium), that stores information that is accessible by aprocessor. Memory 210 may store instructions used in the systems andmethods described herein. While FIG. 2 illustrates a single discretememory 210, the embodiments described herein are not limited to anyparticular arrangement and other embodiments may store information inone combined memory, or with information stored in a differentconfiguration in one or more memories, some local to the othercomponents illustrated in FIG. 2 and/or some shared with, orgeographically located near, other remote computing systems.

As illustrated, memory 210 contains marketing logic 220, which includesan intent logic 222, a value logic 224, and a training module 226.Memory 210 may also contain control logic 240, including one or morealgorithms or models for generally controlling the operation of the datamodeling server 170. These depicted logics may variously represent oneor more algorithms, computational models, decision making rules orinstructions, or the like implemented as software code orcomputer-executable instructions (i.e., routines, programs, objects,components, data structures, etc.) that, when executed by one or moreprocessors 260, program the processor(s) to perform the particularfunctions of theft respective logic. These logics are depicted in FIG. 2as several discrete components, each labelled individually, however, invarious embodiments, the functions of each respective logic may beexecutable on their own or as part of one or more other modules; thatis, any configuration of the depicted logical components may be used,whether implemented by hardware, software, firmware, or any combinationthereof. The capabilities of these various logics are described ingreater detail below.

Memory 210 may also contain raw data 232, aggregate raw data 234, andreal-time data 236, which respectively include one or more databasesstoring information used by intent logic 222, value logic 224, andtraining module 226. It will be noted that while the term “database” or“data storage” may be used herein with reference to elements 210, 232,234, and 236 and/or the data structures therein, these components arenot so limited nor is any particular form, number, or configuration ofdata storage mandated, and any of the described “databases” mayalternatively be an indexed table, a keyed mapping, or any otherappropriate data structure or repository.

The data in raw data 232 is pulled from one or more raw archived datasources, that is, collections of raw data are obtained from web server160. In some embodiments, raw data may also be collected fromthird-party server(s) 180. These raw data collections in raw data 232are illustrated as items 311-316 in FIG. 3 . In an exemplary embodiment,data is pulled from one or more of web server(s) 160 and third-partyserver(s) 180 to populate raw data 311-316 on a scheduled basis, e.g.,every 24 hours, or any other appropriate frequency. In such embodiments,aggregation and/or analysis of such data (described in greater detailbelow) may be performed offline, as the data 311-316 remains unchangedduring the period between updates/refreshes. In other embodiments, anyappropriate schedule may be used or data may be collected in real-time,in response to one or more scheduled events, in cyclical or patternedpull/pushes of data, in alternating, round robin, or variable collectionfrom server(s) 160 and server(s) 180, or by any other configuration. Insome embodiments, servers 160, 180 may be queried on a scheduled basisas to whether any data has been updated and a data pull or data dump maybe performed only if any update has occurred. In some embodiments, theselected method and frequency of data collection may depend on servercomputing resources, transmission latency, frequency of activity in therelevant market, a user base size, seasonality or changeability of data,and/or other relevant considerations.

Raw data 232 includes a variety of information collected as usersinteract with a website or app, make transactions, and the like.Therefore, each of the entries in raw data categories 311-316 can beassociated with a particular user or user device, for example, a user ID(if logged in), a device (by device ID, IP address, MAC address, or thelike), session ID, other information sufficient to identify a user (suchas a unique code or password), or any other appropriate identifyingmechanism. By these means, data from raw data 232 can be aggregated peruser, for a given period of time, to determine cumulative or summarydata for a particular user or set of users that is stored as aggregateraw data 234. While six categories 311-316 of raw data 232 areillustrated, each corresponding to one or more tables (such as hivetables) that can be queried, other embodiments may include any number ofcategories, and any number of data structures. In an exemplaryembodiment, the data between items 311-316 may include some amount ofredundancy, as some user actions may be relevant to several categoriesof data, however in other embodiments, the data is unique to eachcategory 311-316.

Raw data 232 may include landing data 311, that is, an entry point for auser such as a first landing page displayed to a user when accessing awebsite or app. The displayed landing page may be dependent upon a useror user type (e.g., guest, booker, host, first time/repeat visitor,etc.), location, browser/device type, a link or advertisement followed(directing them to a landing page for a certain listing and/or marketingcampaign), a source or referral domain, search engine optimizationtraffic, or any other appropriate mechanism, such data being collectedfrom URL information, analytics services, internal or external webserver logs, and/or other internal or third party systems. Raw data 232may also include search data 312 tracking each search and/or filter orsearch refinement operation made by a user by interacting with one ormore drop down menus, search fields, radio buttons, maps, sliders,point-and-click interfaces, or other user interface components, byfollowing a link or advertisement corresponding to a saved or forwardedsearch, or by entering search information into a local or third partysearch engine. Search data 312 may be pulled from storage in web server160, obtained from a user's referral information, from a characterstring in a URL followed by a user, or otherwise obtained from a thirdparty server 180. In some embodiments, web server logs may containinformation that allow sites to determine what sites referred thevisitor to the present site, what pages a visitor viewed and when, orclient IP addresses (which can roughly approximate location). Dependingon the structure of the URL of the referring site, additionalinformation can be identified (e.g., where a user's search is includedas a character string in a search engine referral addresses). Web serverlogs can also contain information as to which webpages a user hasvisited and the order of a user's traversal through a website. Listingviews data 313 may include all property (or product) listings viewed bya user, e.g., by clicking on a hyperlink or image to review a webpage orpop-up with additional data, and information relating thereto.

Engagement data 314 may include data regarding how users interact with awebsite or application. In an exemplary embodiment, this data couldinclude any of click or view data on links, advertisements, images,listings, pages, and so forth, actions such as liking/disliking,bookmarking, saving, marking as favorite, adding or editing wishlists,scrolling, or other actions (or transactions) made by the user. In someembodiments, engagement data 314 may also include cumulative orcalculated data based on the user's interaction with the website/app,such as an amount of time spent on a single page, listing, or website(or timestamps from which such time can be later determined), eachvisitation or view (from which a frequency of visitation or views can belater determined), referrals made, region(s) of searches and/orbookings, minimum, maximum, and median prices of the properties browsed,or any other relevant collected information, however, in otherembodiments, this cumulative or summary data is calculated by process325 and stored as aggregated raw data 234. Additionally, in someembodiments, engagement data 314 may include third-party data such associal media interaction with or regarding the website or app provider,such as shares, tags, comments, referrals, and the like (such thirdparty data being collected from third-party server(s) 180. In otherembodiments, such actions or data may be included in a separatecategory/type of data stored in raw data 232, such as site activitydata. In some embodiments, engagement data 314 can be collected fromserver logs as users visit pages, from server application software whichgenerates the requested pages, and/or by client software such asJavaScript or mobile applications that can detect user actions (e.g.,hovering over images, watching videos, and/or tracking the amount oftime particular areas of a listing or search results are displayed inthe display area, among other things).

User data 315 may include information associated with a user, e.g., userID, user account information (e.g., name, contact information (e.g.,email address, mailing address, telephone number, date of birth, paymentcard information), a type of user (e.g., guest/registered,business/vacation travelers, etc.), length of membership, bookinghistory, demographic data about the user, such as age, gender, location,language, and device information, GPS, IP address, or location data,operating system, browser, or other device data, user preferences orinterests, connected third party accounts (e.g., social networks) ifapplicable, and the like. Booking data 316 may include bookings andpurchase history of users, including dates and prices and whererelevant, locations, durations of stay, ratings, and/or reviews. Bothuser data 315 and booking data 316 may be obtained from server 160, andin some embodiments, part of such data may be obtained from athird-party server 180.

In the illustrated embodiment, all or subsets of raw data 232 can becollected, processed, parsed, analyzed, and/or aggregated in process325, and the aggregated (and/or otherwise processed data) can be storedas aggregate raw data 234. In one embodiment, theparse/analysis/aggregation in process 325 may involve a standardizationor reformatting of data into more easily searchable or accessible datasets, a process most useful in a case that the data in raw data 232 hasa variety of disparate formats, sizes, or data types. In someembodiments, the aggregate raw data 234 is aggregated by user (e.g., byuser ID, cookie, session ID, location data such as GPS data or check-indata, device ID, MAC addresses, and/or IP addresses), organizingrelevant raw data 232 into sets of user-level data. Put another way,aggregation involves the roll-up of individual actions, transactions,observations, or events (raw data) into summary statistics basedthereon. In this sense, aggregate raw data 234 may be understood aspersonalized or individualized features describing a user's activity ata higher level, though in other embodiments, data may by organized bygroups or sets of users (for example those sharing similarcharacteristics or features, such as location or demographic data).Aggregate raw data 234 may also be organized by user for a certainwindows of time, e.g., 1 day, 7 days, 14 days, or any set timereflective of available data. In some embodiments, certain data such asa user's account or demographic data may not be time-specific but may bestatic, as such information is highly unlikely to change (or changesignificantly) over time, and such information may be included inaggregate raw data 234 even if it is not changed within a recent timeperiod. The time period of such aggregation is limited by the timeperiod of data available in raw data 232.

Aggregate raw data 234 is divided into one or more different tables (orother data structures) at a user-level such as site activity data 322,account data 324, and demographic data 326, each informing user-levelbehavior about usage of the relevant systems of environment 100. Otherconfigurations are possible in other embodiments. For purposes ofillustration, while engagement data 314 of raw data 232 may individuallystore each action (or some subset of actions) made by a user, such asindividual clicks or views of a listing, aggregate raw data 234 maystore, as user-level site activity data 322, patterns of data for theuser over a period of time, for instance, a ratio of listings viewed bythe user in a period of time that remain available, a collection ofmarkets in which the user has search or viewed information, an analysisof saved/liked listings, a computation of a min/max/average of theprices of all the user's viewed listings, and so on.

Information from the raw data sources 232 and from the aggregate rawdata 234 can be gathered as training data 330, and used by the trainingmodule 226 to train data models including intent model 340-1 and valuemodel 340-2. These data models may be later used by intent logic 222 andvalue logic 224 respectively in their real-time calculations. Trainingdata 330 can be collected from both the raw data 232 and the aggregateraw data 234. The training data 330 takes the form of “features”, orsignals fed to the models for training and prediction. Training data330, and the aggregation of data in process 325, is performed on datapreviously stored as raw data 232, that is, the analysis, transmission,processing, and/or aggregation of data is performed on a static set ofdata (e.g., data on past actions) as compared to real-time data, and maytherefore be performed offline (so as not to require a networkconnection via network 130). Potential offline processes are indicatedin FIG. 3 by dashed line arrows. In other embodiments, such processesmay be performed “online” with reliance on a network connection.

Training module 226 is configured to train models that generate one ormore predictive analyses based on the data fed thereinto. Retraining maybe performed continuously or on a scheduled basis for features. In thecase of a home sharing implementation wherein a property listing has anexpected conversion period or “conversion window” in which a user maybook or convert the property (or purchase the product), training module226 inputs into models 340-1 and 340-2 features regarding data relevantto that conversion window (as one example, booking data from bookingswithin a set number of days prior to the day of calculation, the setnumber of days being equal to the conversion window). Upon retraining,additional features are added or joined to training data 330, togenerate new training examples. The most recently added data is used toretraining the models in training module 226 to keep up withseasonality. By retraining the model(s) based on the updated data intraining data 330, a model will automatically change to compensate forseasonal or other time-specific market shifts.

Intent model 340-1 calculates a likelihood that a user will book aviewed property, and value model 340-2 calculates an expected amount(e.g., dollar or monetary amount) at which a user will book, assumingthat they do book. The trained value models 340-1 and 340-2 are used, byintent logic 222 and value logic 224 respectively, to obtain a useracquisition score representing the return on investment that would bereceived for marketing to the user.

The value model 340-2 accepts as input a set of features about the userand outputs a prediction of the monetary value (i.e., a base price) of aproperty booking that a user will make within a certain period of time,referred to herein as a “conversion window” or “conversion time”, fromthe date that the user viewed the listing for the property. The valuemodel 340-2 functions under an assumption that the user will in factmake a booking during that conversion window, that is, that theprobability of booking is 1. The prediction performed by the value model340-2 is based on one or more machine learning algorithms. In oneembodiment, the value model uses multiple linear regression to predictuser behavior, though in other embodiments, value model 340-2 may tunethe signal weights in accordance with another type of supervisedregression or classification algorithm. In an exemplary embodiment, inorder to calculate the expected booking value of a property, the valuemodel 340-2 may use the following equation:

V=V(booking value|booking,person)=Σ₀ ^(m) w _(i) *x _(i)   (Equation 1)

where V is the booking value, m is the number of features, w_(i) is theweight of each feature i, and x_(i) is the value of feature i. Tocalculate the value of the feature weights w_(i), a subset of featuredata is obtained regarding users who booked or purchased within thethreshold conversion window from the date the user views the list (insome embodiments, a first viewing, and in others, a most recentviewing). In an exemplary embodiment, the calculation is performed as agradient descent based on the absolute value of the difference betweenthe predicted value and the actual value, the difference being used toadjust the weightings. Training module 226 can sort the features by theabsolute value of the weights to identify the most relevant features foreach model.

In the exemplary embodiment, value model 340-2 weighs the followingfeatures most heavily: destination city and median price of listingsviewed by the user within a set window (e.g., 1 day, 7 days, 14 days,etc.). In an exemplary embodiment, the value model determines the mostapplicable window, based on the aggregated data. As just one example,where highly relevant user activity occurs prior to, e.g., 7 days beforethe most recent data pull, the value model may use a time window with agreater number of days. Other embodiments may default to a predeterminedwindow of time, e.g., 14 days, to the largest window of time for whichdata is available, to a relatively small period of time (e.g., 1 day)where historical user or market data suggests a high rate of activity,or to any other appropriate window. The same period of time is appliedby the value model in its calculation for different users, therebyallowing like-for-like comparison and aggregation of user data.

In some embodiments, other features relating to the price of listings(e.g., min, max, or average price viewed by the user, prices of listingsrelevant to the locality of the user's search, prices of the user'shistorical views or bookings, price changes of listings depending onseasonality, etc.) may also or alternatively be weighted heavily. In oneexample embodiment, price-related features, including destination city,and other price-related market factors, may be weighted so much morethan other features that the effect of other types of features on theoutput of the value model become negligible, however, other combinationsof weights are possible in different embodiments. Of course, the valuemodel 340-2 is not limited to consideration of the features above, andin some embodiments, there is no limit on potentially relevant features.Indeed, such features and the weightings thereof may change based onseasonality, on market, on user type, etc.

Any number of features may be used by the value model, where theweighting of features may differ based on the data generated by thetraining module 226. That is, in the execution of value model 340-2 bytraining module 226, the training data 330 is used to determine a set ofweights w_(i) with which probability to book is optimized for theparticular user or set of users. Those determined weights w_(i) are thenused in the real-time application of value model 340-2 by value logic224 to the aggregated and real-time data 234, 236 to generate an actualprediction.

As described above, the value model 340-2 employs a gradient descentbased on the absolute value of the difference between the predictedvalue and the actual value. Gradient descent is an iterative processthat works by adjusting the weight(s) of a model looking to minimizeerror, where the error may be calculated in a variety of ways. In anexemplary embodiment, the value model 340-2 is trained to minimize theabsolute value of the difference between the predicted price value andthe actual price value. For purposes of example, FIGS. 4A-E illustratesingle variable linear regression, where the value model 340-2 attemptsto predict the dependent variable, mapped to the y-axis, here the valuethat the user will pay for the booking, against an independent variable,mapped to the x-axis, here corresponding to a feature from theaggregated raw data. In the illustrated example, the independentvariable is a median viewed price (the median price of listings viewedby the user), however, any appropriate feature may be used in differentembodiments. The calculation begins with an initial guess as to thevalue, and based on that guess, an error is calculated by comparingactual results to the predicted results. The error and the inputvariables are then used to make an adjustment to the weights, and theprocess repeats until the error reaches a suitable minimum value. Theamount of adjustment to the weights can be controlled by momentum andlearning rate modifiers. These modifiers may be constant or may vary asthe process goes on.

With reference to FIGS. 4A-F, the output of the linear predictionfunctions are shown as lines 410-418 of a scatterplot. A sample set oftraining parameters are shown as dots (e.g., dot 401), which maintainthe same placement on each of FIGS. 4A-4F. As shown in FIG. 4A, thelinear prediction function 410 starts off as an initial guess with azero sloped line 410. In other embodiments, the initial guess could alsobe a near vertical line or a line with a slope based on previoustraining. Training samples are processed through the training algorithmand the collective error is used to make corrections to the slope andintercept of the line, where the resulted linear prediction functionchanges from line 410 (FIG. 4A) to line 412 (FIB. 4B), then to line 414(FIG. 4C), and finally to line 416 (FIG. 4D), where the error from thetraining parameters falls below a minimum value. In some embodiments,the training data may be processed multiple times until a local minimumin the error function is located. FIG. 4E illustrates an exemplarycalculation of error by summing the vertical difference (shown as avertical dotted line) between an actual value for a series of points(the value y of the training parameter with independent variable x,located at the black dots) and the predicted value (the value y′ on line416 corresponding to independent variable x, located at the white circleon line 416).

The same principles apply generally to multiple linear regression, withthe exception that the prediction comes in the form of an n−1dimensional object, where n is the number of variables (viz., features)for the value model 340-2. The embodiment of FIG. 4F uses twoindependent variables (feature 1 and feature 2) and one predictionfunction 318, though any number of features may be used in differentembodiments. Linear regression is performed for each feature, resultingin functions shown by solid line 416 (feature 1, median viewed price)and dashed line 418 (feature 2, destination city). The relevance of eachfeature (its correlation with booking value) is calculated through thesame process described with reference to FIGS. 4A-4E, and the mostfeatures most relevant to value are assigned higher weights than thoseless relevant to value. Through the assignment of weights to thefeatures in training data 330, the best fitting value model 340-2 can bedetermined.

In an exemplary embodiment, intent model 340-1 works in a very similarway. The intent model 340-1, when being trained in training module 226,accepts as input a variety of features about the user from training data330. In one embodiment, the features input into intent model 340-1 arethe same as the features input into value model 340-2, though in otherembodiments, different sets of features may be used. The intent model340-1 outputs a predicted probability as to whether the user will book aproperty (or, in other embodiments, purchase a product) within a certainperiod of time, i.e., within the “conversion window”, from the date thatthe user viewed the listing for the property. In the exemplaryembodiment, intent model 340-1 weighs all or a subset of the followingtypes of features most heavily:

-   -   1) User account history (including, e.g., a guest classification        into a visitor, user, or booker type, a number of previous        bookings, number of days since the visitor created their        account, number of day since the user first booked);    -   2) Past or historical site activity (e.g., a frequency of viewed        listings within a set period of time, such as 1, 7, or 14 days);    -   3) Same-day or recent site activity, obtained from real-time        data sources 236 (e.g., whether and how frequently the user        logged in, a number of searches, a number of listing views,        contacts, and/or bookings, a number of seconds on the site);    -   4) Listing availability (e.g., a ratio of listings available on        the desired reservation date(s) and total viewed listings within        a set period of time, such as 1, 7, or 14 days);    -   5) An origin location of a user (based, e.g., on user-provided        locations, GPS, IP address, or location data, operating system,        browser, or other device data); and    -   6) A destination location in which the user has searched.

In some embodiments, other features reflecting, summarizing, oranalyzing a frequency of user activity may also or alternatively beweighted heavily. Of course, the intent model 340-1 is not limited toconsideration of the features above, and in some embodiments, there isno limit on potentially relevant features. Indeed, such features maychange based on seasonality, on market, on user type, etc. In oneexample embodiment, features types (1)-(6) above may be weighted so muchmore than other types of features, that the effect of other types offeatures on the output of the value model become negligible, however,other combinations of weightings are possible in different embodiments.The set period of time used by the intent model 340-1 is, in anexemplary embodiment, determined by the intent model based on theaggregated data. Other embodiments may default to a predetermined windowof time, e.g., 14 days, to the largest window of time for which data isavailable, to a relatively small period of time (e.g., 1 day) wherehistorical user or market data suggests a high rate of activity, or anyother appropriate window. The same period of time is applied by theintent model in its calculation for different users, thereby allowinglike-for-like comparison and aggregation of users' data.

The intent model 340-1 uses a methodology similar to that of the valuemodel, but instead of a linear relationship between the independentvariable and the predicted variable (as in the value model 340-2), theintent model 340-1 may use a logistic relationship. Exemplary FIG. 5Ashows an S-shaped curve 510 such as a sigmoid or arctangent used insteadof a linear equation. In an exemplary embodiment, the intent model 340-1may use the following equation:

$\begin{matrix}{P = {{P\left( {booking} \middle| {person} \right)} = \frac{1}{1 + e^{- {\sum_{0}^{n}{\omega_{i}*x_{i}}}}}}} & \left( {{Equation}2} \right)\end{matrix}$

As depicted in FIG. 5B, during the regression process, the logisticprediction function 510 can be adjusted in different ways such as movingit to the left (512) or right (514) or changing the general shape (514)in such a manner so as to reduce the error to a minimum (i.e., toconform the shape of the curve to the actual scattered data points). Inan exemplary embodiment, the dependent variable of the intent model340-1 is a binary value for booking intent, either true or false (e.g.,either the user made a booking or they did not, represented by the dots502 and 504, respectively). For a threshold classification level of thecurve (e.g., 0.5), the points below are classified as not likely tobook, and points above are classified as likely to book. Theclassification threshold may be set by computing points of a receiveroperating characteristic curve of true/false positive bookingpredictions (from training data 330), and an area under the curve (AUC)metric can be applied to the receiver operating characteristic (ROC)curve to output a rate of successful classification by the logisticmodel. The classification threshold maximizing successful classificationmay thereafter be selected. The output of intent model 340-1 is aprobability of any positive value ranging from 0 to 1, where 0 indicatesthat the user will not book, and 1 indicates that they will definitelybook.

In some embodiments, models 340-1 and 340-2 can be specifically trainedwith fewer features, for example, if an advertiser's advertisementcampaigns only allows for the targeting of users along a specific set offeatures, the value model and/or the intent model can be trained usingjust those features. In another embodiment, the model can be run usingmultiple combinations of features and averaging or summing the resultsrelative to other information available. In still other embodiments,different intent models and/or value models may be available fordifferent marketing purposes (e.g., different campaigns, markets,advertising services, etc.) such that the feature sets available foradvertisement targeting may match the features available for thepredictive models. In some embodiments, for real-time requests includingpage loads as in the case where the user visits the website and thelanding page rearranges and selects items to display as a result of theintent, value, and expected value predictions data modeling server 170may prioritize speed of calculation. In such embodiments, instead ofusing all possible features, an optimized list of the most relevantfeatures that can be used by an optimized model can be generated toprocess real-time requests. The optimized model is trained using thesame data set but only used the most relevant subset of features asinput. Various feature values are connected by various weights w₁−w_(m)to their respective predictive model.

Through the application of these training techniques, the intent model340-1 can be applied by intent logic 222 to calculate a likelihood 362that a user will book a viewed property, and value model 340-2 can beapplied by value logic 224 to calculate an expected price 364 (a dollaror monetary amount) at which a user will book, assuming that they dobook. The real-time application of intent model 340-1 by intent logic222 and of value model 340-2 by value logic 224 is performed not on thetraining data 330 but on a set of information collected from aggregateraw data 234 and real-time data 236. Real-time data 236 may beunderstood as a collection of data that has accumulated temporally afterthe last collection of data of raw data 232 from the data sources onservers 160, 180, that is, recent data that could not have been capturedin the previous data dump/data pull. In the illustrated embodiment, suchrecent data includes users' site activity data 352 (the interactions ofusers with the website or app, collected in real time from web server160), however in other embodiments, real-time data in any of the sametypes as data 311-316 may be collected. As the collection (pull or dump)of data in real-time requires a connection to networked data sources(online stream services) over network 130, such data transmission mustbe done “online.” A networked, or “online” process is illustrated inFIG. 3 as a solid line arrow.

In one embodiment, by multiplying the results 362, 364 of the valuemodel and intent model together, data modeling server 170 can obtain auser acquisition score representing the a dollar value return oninvestment that would be received for marketing to the user thataccounts for user intent. The function can be set up where the expectednumber and features (e.g., demographics, user history, etc.) of thetarget audience can be entered into the model and the resultssummed/analyzed to predict the expected reach and expected value (e.g.,return on investment) of a particular advertisement campaign. Throughthe use of standardized feature labels, shared schema, or by following aspecified protocol, the models can be used by the advertiser to makedeterminations, marketing plans, to determine expected penetration orvalue of the advertisements, to suggest different campaigns, to displaycontent, to help identify target users for an advertisement orpromotion, or for other purposes.

An exemplary process is illustrated in FIG. 6 . The process begins atStep 602, at which raw data regarding user activity is obtained from avariety of networked sources. In some embodiments, the raw data mayinclude any of: user landing data, user search data, user views data,historic/past user engagement or activity data, user account and/ordemographic data, and user booking data, though other data (or anysubset of this data) may be used in other embodiments. In Step 604, thisraw data is summarized or aggregated into user-level aggregate data,specific to a user for a set time period (1 day, 1 week, 1 month, etc.).The aggregate data includes a variety of features, or summary-levelstatistics, for the user. All features are extracted, in Step 606, for aconversion window, i.e., a window of time (generally a number of days)defining a period between the time that a user has viewed a product orproperty listing and the time at which the user would be expected topurchase or reserve the product, if the user were going to do so.

A machine learning model to calculate user intent, referred to as anintent model, is trained on the extracted features in Step 608. Theintent model assigns a set of weights to the extracted features, withfeatures relevant to frequency of user action (e.g., product/propertylistings most viewed by the user, latest user logins) being weighted thehighest. In some embodiments, other highly-weighted features include oneor more of those related to: user account history, usersame-day/real-time site behavior, listing/product availability, userorigin location, destination of viewed products/listings, and userpast/historic site activity/behavior, though other data (or any subsetof this data) may be used in other embodiments. Similarly, a machinelearning model to calculate expected value of a purchase/booking if oneis made, referred to as an value model, is trained on the extractedfeatures in Step 610. The value model assigns a set of weights to theextracted features, with features relevant to price (e.g., destinationcity/location, and an average/median price of product/property listingsviewed by the user within the time window) being weighted the highest.In some embodiments, other highly-weighted features include one or moreof those related to: min/max/avg price viewed by the user, price(s) oflistings relevant to the locality of the user's search, prices of theuser's historical views or bookings, price changes of listings dependingon seasonality, etc., though other data (or any subset of this data) maybe used in other embodiments.

The intent model and value model are continuously re-trained, such thatwhen there is updated raw data available in Step 612 (occurring, in anexemplary embodiment on a set schedule, such as every 24 hours), theprocess cycles back to Step 602, with new information from the lastupdate window being added to the training examples, and the respectiveweightings of the intent model and value model being recalculated inSteps 604-610 in view of the updated information.

As additional user activity may have occurred since the last update ofraw data, Step 614 involves obtaining data in real-time from one or morereal-time streaming sources. In some embodiments, this data may beuser-level real-time data, that is, organized so as to provideuser-level statistics, however in other embodiments, the format of thereal-time data may be similar to that of the raw data. The real-timedata may include, in an exemplary embodiment, all recent user activityon a website, including product/property views, clicks, saved,favorites, etc. In other embodiments, other types of recent information,such as user account data (or other raw data) may be pulled inreal-time.

In Step 616, the real-time data and the user-level aggregate data may beentered into the previously-trained intent model. The intent model,applying the set of weightings of the features as calculated in Step608, calculates a probability that the user will make a purchase (e.g.,book a property). In some embodiments where user conversion isprioritized regardless of dollar value of return, no monetary return oninvestment (ROI) may be necessary (NO in Step 618), and the userprobability of booking determined by the intent model may be sufficientto generate a marketing plan, as in Step 624. In such embodiments, thegenerated marketing plan may be used in Step 626 to determine a value(e.g., dollar or resource value) to invest in marketing to the user,such as an advertising bid, a maximum marketing cost, or the like. Inembodiments where a monetary ROI (the expected value from the user) isthe target calculation (YES in Step 618), the process continues to Step620, in which the value model is applied using the set of weightings ofthe features as calculated in Step 610. The value model outputs anexpected value under an assumption that the user will definitely make apurchase/booking. This value may be multiplied with the intent to bookcalculated by the intent model, in Step 622, to determine the ROI forthe user, factoring in both the user intent and the booking value.

By these means, a highly personalized ROI determination can be made fora user that maximizes revenue, uses each visitor's past site activities,account history, and demographic information. This determination isused, in Step 624 to generate a related marketing plan, and in Step 626to determine a marketing investment (e.g., an advertising bid) for theuser. In Step 628, these determinations can be scaled over multipleusers to campaign-level statistics that take into considerationcustomer-specific data, so as to obtain information including thepotential conversion rates and expected values of potential salesgenerated by various potential advertisement campaigns. In order toperform the scaling of Step 628, the ROI of each of a set of users(Steps 602-624) is evaluated, the set of users sharing, for example, oneor more common features such as geographic origin location of the user,geographic location of the market, price range of viewed listings, etc.As one example, if a marketing campaign is to be launched in aparticular location, such as San Francisco, Calif., the processes ofFIG. 6 may be applied to see, in aggregate, the likelihood of conversionand/or the monetary return on investment, of every user located in SanFrancisco, allowing campaign-level decisions to be made.

In one embodiment, the calculation performed by one or both of theintent model and the value model can be used to determine an expectedmarket penetration and/or evaluate return on investments of proposedadvertisement campaigns. For instance, an output of the intent modeland/or the value model may be used to determine whether a user iseligible for a discount or coupon, and/or the frequency or value of suchdiscount. In another embodiment, the calculation performed by one orboth of the intent model and the value model can be used to selectmarketing targets or advertisements to display or deliver to potentialcustomers. For instance, the content and quality of an advertisement maybe calculated in correlation with a calculated intent score, e.g.,higher quality of advertisements or greater total spend may be focusedon consumers with lower calculated intent, under the assumption thathigh-intent consumers do not need additional invested resources to beinduced to convert.

In still another embodiment, the calculation performed by one or both ofthe intent model and the value model can be used to select informationto be presented on web-based user landing pages including what contentto display and how to order or arrange various elements (in someembodiments, property listings, image or text content, recommendations,etc.). As one such example, where the intent model indicates a consumerwith a high-intent to book a property listing, the consumer can bedirected to a landing page showing other listings with similar features,while, a consumer with a lower calculated intent score may be directedto a landing page with more exploratory results, relating to a point ofinterest, etc. Of course, other applications may be possible in otherembodiments.

In one embodiment relating to targeting marketing activity, one or moreof the intent model and the value model exclude from their estimationsdata relating to users with comparatively low intent to purchase and/orcomparatively high intent to purchase (e.g., the lowest and highest 25%,or any appropriate percentage or number of users), under the assumptionthat targeted marketing is unlikely to alter the intent of such users.This exclusion may result in higher incrementality (a higher lift inconversion).

The systems and methods described herein are not limited to marketsrelating to property rental/booking, but can be used for a variety ofmarkets or a variety of purposes in the home rental market. Forinstance, by changing the target variables from those related toproperty-specific characteristics such as, e.g., destination city, thedescribed systems can be extended to optimize for other metrics. In oneembodiment, the data modeling server can be used to determine metricsrelated to mobile app install, and the intent model and/or value modelmay be used to use to determine a likelihood a user will install an appand/or the price they would pay to do so, respectively. In otherembodiments, metrics related to user sign ups, consumer lifetime valuecalculations, and the like can be determined.

The foregoing is merely illustrative of the principles of thisdisclosure and various modifications may be made by those skilled in theart without departing from the scope of this disclosure. The abovedescribed embodiments are presented for purposes of illustration and notof limitation. The present disclosure also can take many forms otherthan those explicitly described herein. Accordingly, it is emphasizedthat this disclosure is not limited to the explicitly disclosed methods,systems, and apparatuses, but is intended to include variations to andmodifications thereof, which are within the spirit of the followingclaims.

As a further example, variations of apparatus or process parameters(e.g., dimensions, configurations, components, process step order, etc.)may be made to further optimize the provided structures, devices andmethods, as shown and described herein. In any event, the structures anddevices, as well as the associated methods, described herein have manyapplications. Therefore, the disclosed subject matter should not belimited to any single embodiment described herein, but rather should beconstrued in breadth and scope in accordance with the appended claims.

The following is claimed:
 1. A method for determining a predicted value of a user purchase via a computer system, the method comprising: storing a first plurality of user metrics in a raw data format, the first plurality of user metrics relating to a user's activity on the computer system; storing a second plurality of user metrics associated with the user's activity on the computer system within a window of time, the second plurality of user metrics being generated by aggregation of data in the first plurality of user metrics; grouping the second plurality of user metrics into one or more features, each feature corresponding to a statistic of the data in the second plurality of user metrics; training a first machine learning model, using a first regression-based algorithm, based on a first set of feature-specific weighting that distinguishes between the one or more features; training a second machine learning model, using a second regression-based algorithm, based on a second set of feature-specific weighting that distinguishes between the one or more features; generating a predicted probability of the user purchase, based on (a) the second plurality of user metrics and (b) the first machine learning model; and generating a predicted return on investment (ROI) value, based on (a) the second plurality of user metrics, (b) the second machine learning model, and (c) the generated predicted probability of the user purchase.
 2. The method of claim 1, wherein the training of the first machine learning model comprises identifying features, of the one or more features, relevant to the user's past activity on the computer system, and wherein the training of the second machine learning model comprises identifying features, of the one or more features, relevant to the price of the user purchase.
 3. The method of claim 1, wherein the training of the first machine learning model comprises identifying features, of the one or more features, relevant to a frequency of the user's activity on the computer system.
 4. The method of claim 1, wherein generating the predicted ROI value further comprises: determining, based on (a) the second plurality of user metrics and (b) the second machine learning model, the expected value of a purchase if made by the user; and calculating the predicted ROI value by multiplying the expected value of the purchase by the predicted probability of the user purchase.
 5. The method of claim 1, wherein the first regression-based algorithm is a logistic regression model, and the second regression-based algorithm is a linear regression model.
 6. The method of claim 1, wherein the user purchase relates to a booking of a property listing viewed by the user via the computer system; and wherein the first set of feature-specific weightings weighs features corresponding to one or more of the following items more heavily than features that do not correspond to one or more of the following items: a) user account history, b) past or historical user activity, c) recent user activity, d) availability of a set of property listings viewed by the user, e) location of the user, and f) location of the set of property listings viewed by the user.
 7. The method of claim 1, wherein the user purchase relates to a booking of a property listing viewed by the user via the computer system; and wherein the second set of feature-specific weightings weighs features corresponding to one or more of the following items more heavily than features that do not correspond to one or more of the following items: a) location of a set of property listings viewed by the user, and b) a median price of the set of property listings viewed by the user.
 8. A system for obtaining a real-time computative prediction of a return on investment (ROI) value for a user based on user data provided by one or more servers over a network, the system comprising: a web server; a memory configured to store a plurality of user data in a raw data format, the plurality of user data relating to a user's activity on the system, the plurality of user data being obtained from the web server; and at least one processor configured to: transform the plurality of user data into a set of aggregated user metrics associated with the user's activity on the system within a window of time; store the aggregated user metrics in the memory; categorize the aggregated user metrics into one or more training signals; generate a first set of weighted multipliers for the one or more training signals using a first regression-based algorithm; train a first machine learning model based on the first set of weighted multipliers; generate a second set of weighted multipliers for the one or more training signals using a second regression-based algorithm; train a second machine learning model based on the second set of weighted multipliers; apply the first machine learning model to the aggregated user metrics to calculate a predicted probability that the user will make a purchase; apply the second machine learning model to the aggregated user metrics to calculate a predicted value of the purchase; and generate the prediction of the ROI value for the user based on the predicted probability that the user will make the purchase and the predicted value of the purchase.
 9. The system of claim 8, wherein the at least one processor is further configured to: obtain real-time data relating to the user's activity on the system; and store the real-time data in the memory, wherein the first machine learning model is applied to (a) the aggregated user metrics and (b) the real-time data to calculate the predicted probability, and wherein the second machine learning model is applied to (a) the aggregated user metrics and (b) the real-time data to calculate the predicted value.
 10. The system of claim 8, wherein generating the first set of weighted multipliers comprises identifying training signals relevant to the user's past activity on the system.
 11. The system of claim 8, wherein generating the first set of weighted multipliers comprises identifying training signals relevant to a frequency of the user's activity on the system.
 12. The system of claim 8, wherein generating the second set of weighted multipliers comprises identifying training signals relevant to a purchase price of the purchase.
 13. The system of claim 8, wherein the first regression-based algorithm is a logistic regression model, and the second regression-based algorithm is a linear regression model.
 14. The system of claim 8, wherein the first set of weighted multipliers weighs training signals corresponding to one or more of the following items more heavily than training signals that do not correspond to one or more of the following items: a) user account history, b) past or historical user activity, c) recent user activity, d) availability of a set of property listings viewed by the user, e) location of the user, and f) location of the set of property listings viewed by the user.
 15. The system of claim 8, wherein the second set of weighted multipliers weighs training signals corresponding to one or more of the following items more heavily than training signals that do not correspond to one or more of the following items: a) location of a set of property listings viewed by the user, and b) median price of the set of property listings viewed by the user.
 16. A method comprising: storing, in a memory, user data relating to a user's activity on a computer system; storing, in a memory, user metrics associated with the user's activity on the computer system within a window of time, the user metrics being generated by aggregating a set of data of the stored user data; identifying a first set of target variables directed to optimizing user acquisition; training a first machine learning model to generate a predicted probability of a user action, using a first regression-based algorithm, based on the first set of target variables; identifying a second set of target variables directed to optimizing revenue resulting from the user action; training a second machine learning model, using a second regression-based algorithm, based on the second set of target variables; generating the predicted probability of the user action, based on the application of the first machine learning model to the user metrics; generating a predicted return on investment (ROI) value, based on the predicted probability of the user action and the application of the second machine learning model to the user metrics.
 17. The method of claim 16, further comprising: obtaining real-time data relating to the user's activity on the computer system, wherein the predicted probability of the user action is generated based on the application of the first machine learning model to (a) the user metrics and (b) the real-time data, and wherein the predicted ROI value is generated based on the application of the second machine learning model to (a) the user metrics and (b) the real-time data.
 18. The method of claim 16, wherein the first set of target variables directed to optimizing user acquisition relate to a frequency of the user's activity on the computer system.
 19. The method of claim 16, wherein the user action is the booking of a property rental via the computer system, and wherein the target variables directed to optimizing revenue relate to a price of the property rental. 